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Abstract 


In  their  September  2015  AF  Future  Operating  Concept,  the  Secretary  of  the  Air  Force 
and  the  Chief  of  Staff  of  the  Air  Force  called  for  the  identification  of  critical  thinkers  and  metrics 
to  track  and  measure  critical  thinking  (CT)  skills.  To  answer  their  charge,  this  quantitative 
research  project  used  the  Watson-Glaser  Critical  Thinking  Appraisal  to  establish  a  baseline  of 
current  CT  skills  among  active  duty  Air  Force  (AF)  officers  attending  professional  military 
education  (PME)  in-residence  at  Air  Command  and  Staff  College  (ACSC),  School  for  Advanced 
Air  and  Space  Studies  (SAASS),  and  Air  War  College  (AWC).  T-tests  indicated  no  statistically 
significant  difference  in  the  CT  skills  of  the  sample  of  ACSC  and  AWC  students,  but  did  reveal  a 
significant  difference  between  ACSC  and  SAASS  students.  Furthermore,  the  average  raw  score 
for  ACSC  and  AWC  was  at  the  36th  percentile,  with  SAASS  at  the  61st  percentile,  when 
compared  to  the  normative  group  with  a  Master’s-level  education.  This  result  established  AF 
officers  attending  ACSC  and  AWC  in-residence  were  below  average  in  CT  skills  when 
compared  with  individuals  at  the  same  academic  level.  These  findings  suggested  that  the 
operational  leadership,  and,  most  importantly,  levels  of  PME  and  education,  fail  to  develop  the 
CT  skills  of  AF  officers.  As  a  partial  solution  to  this  problem,  this  paper  will  make 
recommendations  for  modifications  in  the  delivery  of  PME  curriculum. 
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Introduction 


In  the  September  2015  Air  Force  Future  Operating  Concept  (AF  FOC),  the  Secretary  of 

the  Air  Force  (SecAF)  and  the  Chief  of  Staff  of  the  Air  Force  (CSAF)  identified  the  need  for: 

...Airmen  who  display  critical  thinking  in  complex  situations,  are  educated  and  trained 
appropriately,  and  ultimately  are  empowered  and  trusted  to  execute...  This  foundation  is 
built  by  recruiting  Airmen  with  indicated  potential  for  critical  thinking  and  adaptive 
behavior;  screening  for  these  attributes  will  require  new  metrics  and  forms  of  evaluation.1 

However,  no  published  or  publically  available  data  exists  to  address  (1)  the  current  state  of 

critical  thinking  (CT)  skills  in  the  Air  Force  (AF),  (2)  a  recommended  metric  by  which  to 

measure  CT  skills,  and  (3)  whether  the  existing  state  of  CT  skills  satisfies  the  AF  FOC’s  intent. 

Using  the  Watson-Glaser  Critical  Thinking  Appraisal  (WGCTA),  this  research  addressed  these 

points  by  focusing  on  the  active  duty  (AD)  AF  students  attending  Air  Command  and  Staff 

College  (ACSC),  School  for  Advanced  Air  and  Space  Studies  (SAASS),  and  Air  War  College 

(AWC)  in  Academic  Year  2016  (AY16).  These  three  populations  within  Air  University  (AU),  by 

virtue  of  the  developmental  education  boarding  process,  provided  a  representative  sample  of  the 

top  20%  of  AD  AF  officers  for  AY  16. 2  The  research  explored  the  state  of  CT  skills  as  an 

indicator  for  SecAF  and  CSAF,  not  whether  or  not,  or  how  much,  AU  integrated  CT  into  the 

curriculums. 

Since  before  1997,  the  AF  has  identified  CT  as  a  key  skill,3  yet  the  AF  has  not 
established  any  metrics  to  provide  a  baseline  assessment  of  CT  in  the  officer  corps.  Several  AF 
studies  identified  the  need  for  CT,  but  the  authors  limited  the  recommendations  to  ways  to 
improve  CT  programs  without  first  assessing  the  state  of  CT  skills. 4  This  foundational  study, 
through  a  quantitative  methodology,  provided  a  baseline  assessment  of  CT  skills  from  a  sample 


of  ACSC,  AWC,  and  SAASS  students  in  AY16. 


Thesis 


This  foundational  study  used  the  WGCTA  to  measure  the  CT  skills  of  a  sample  of  AD 
AF  attending  ACSC,  SAASS,  and  AWC  in  order  to  establish  the  current  baseline  of  CT  as 
represented  by  the  top  20%  of  AD  AF  officers  in  AY16.  The  research  design  tested  the 
hypothesis  that  there  was  no  statistically  significant  difference  in  the  CT  skills  of  IDE  and  SDE 
students.  SAASS,  as  a  more  selective  advanced  studies  program,  provided  an  additional  data 
point  for  comparison  with  ACSC  and  AWC.  The  research  answered  the  following  four  research 
questions: 

•  What  was  the  current  state  of  CT  skills  as  measured  by  the  WGCTA? 

•  Using  t-tests,  were  there  any  significant  differences  between  all  three  schools? 

•  How  did  the  sample’s  performance  compare  with  a  graduate  degree  normative  group? 

•  What  CT  instructional  methods  could  Air  University  apply  to  in-residence  PME? 

The  results  of  this  research  provided  a  starting  point  for  data-driven  decision  making 

regarding  the  integration  of  CT  into  PME  as  well  as  the  operational  AF  in  order  to  meet  the  AF 
FOC’s  requirements. 

Literature  Review 

As  identified  in  AF  Doctrine  Document  1-1,  senior  leaders  expect  Airmen  to  think 
critically:  “education  provides  critical  thinking  skills,  encouraging  exploration  into  unknown 
areas  and  creative  problem  solving.  Its  greatest  benefit  comes  in  unknown  situations  or  new 
challenges;  education  prepares  the  individual  for  unpredictable  scenarios.”5  The  National 
Military  Strategy  (NMS)  required  the  Department  of  Defense  (DoD)  to  update  how  it  “selects 
and  incentivizes  faculty,  rewards  critical  thought,  and  promotes  our  most  innovative  minds. 
Continuous,  demanding  education  inspires  new  ideas  and  identifies  better  ways  to  accomplish 
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our  mission.”6  Furthermore,  the  NMS  identified  two  out  of  six  attributes  of  the  joint  leader  to  be 
“Think  critically  and  strategically  in  applying  joint  warfighting  principles  and  concepts  to  joint 
operations”  and  “Anticipate  and  adapt  to  surprise,  uncertainty,  and  chaos.”7  The  Planner’s 
Handbook  for  Operational  Design  further  highlights  CT  “is  instrumental  to  a  sufficient 
understanding  of  the  operating  environment.”8  CT  is  not  a  panacea  for  the  volatile,  uncertain, 
complex,  and  accelerating  global  environment,  but  it  provides  a  framework  to  improve  analytic 
rigor  and  reduce  the  number  of  flawed  decisions. 

Senior  leaders  in  the  AF  and  DoD  frequently  emphasized  the  need  for  CT,  but  they  rarely 
provided  any  refined  directives  defining  CT  skills  or  how  these  skills  should  be  measured  and 
delivered.  This  leaves  implementation  to  either  Air  University  or,  for  those  not  selected  to  attend 
in-residence  IDE  or  SDE,  the  individual,  with  limited  tools  for  execution.  The  following  section 
details  the  challenges  of  defining  the  construct  of  CT,  presenting  a  consensus  that  CT  skills  (1) 
are  the  product  of  a  personal  and  life-long  dedication  to  improving  the  accuracy  and  logic  of 
thought  patterns  and  (2)  can  be  both  taught  and  measured.  Based  on  a  comparison  of  CT 
development  programs  in  academic  and  business  settings,  the  deliberate  development  of  CT 
skills  in  both  PME  and  throughout  the  operational  AF  would  be  possible  to  implement. 

Concept  of  Critical  Thinking 

Definitions  of  CT  range  from  abstract  constructs  to  specific  measurable  skills.9  The 
National  Council  for  Excellence  in  CT  (NCECT)  approached  the  definition  with  two 
components:  “(1)  a  set  of  information  and  beliefs  generating  and  processing  skills,  and  (2)  the 
habit,  based  on  intellectual  commitment,  of  using  those  skills  to  guide  behavior.”10  In 
comparison,  Paul  and  Elder  defined  CT  as:  “the  art  of  analyzing  and  evaluating  thinking  with  a 
view  to  improving  it.”11  Vaughn  provided  a  succinct  working  definition  for  the  construct  of  CT: 
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“the  systematic  evaluation  or  formulation  of  beliefs  or  statements,  by  rational  standards.”12 

Watson  and  Glaser,  the  creators  of  the  survey  instrument  used  in  this  study,  viewed  CT  as: 

. .  .a  composite  of  attitudes,  knowledge,  and  skills.  This  composite  includes:  (1)  attitudes 
of  inquiry  that  involve  an  ability  to  recognize  the  existence  of  problems  and  an 
acceptance  of  the  general  need  for  evidence  in  support  of  what  is  asserted  to  be  true;  (2) 
knowledge  of  the  nature  of  valid  inferences,  abstractions,  and  generalizations  in  which 
the  weight  or  accuracy  of  different  kinds  of  evidence  are  logically  determined;  and  (3) 
skills  in  employing  and  applying  the  above  attitudes  and  knowledge.13 

While  even  this  small  sample  of  available  CT  definitions  provides  additional  and  valuable 

insight,  the  focus  remains  on  a  systematic  evaluation  of  an  individual’s  thoughts  by  rational 

standards.  Although  simplistic,  Vaughn’s  definition  provides  the  best  balance  between  the  scope 

of  the  concept  and  being  sufficiently  succinct  for  use  in  everyday  discussions  around  the  AF. 

With  this  foundation  for  the  concept  of  CT,  one  can  identify  skills  with  more  specificity 

for  purposes  of  direct  comparison.  As  tested  in  the  WGCTA,  Watson  and  Glaser  delineated  the 

five  skills  of  CT:  inference,  recognition  of  assumptions,  deduction,  interpretation,  and  evaluation 

of  arguments  (see  Table  1). 

Measuring  Critical  Thinking 

Researchers  have  dedicated  decades  of  study  on  various  methodologies  measuring  CT. 
While  there  are  somewhat  intrusive  and  time-intensive  methods  where  an  individual  has  a  one- 
on-one  examination  with  a  trained  evaluator,  most  researchers  and  organizations  use 
standardized  assessment  instruments.  Although  multiple  CT  tests  are  available,14  the  WGCTA 
was  the  most  effective  instrument  to  answer  the  proposed  hypothesis  and  research  questions. 

The  WGCTA  is  computer  administered  and  has  established  validity  and  reliability,  as  well  as 
normative  groups  based  on  a  wide  range  of  populations.15  The  WGCTA  assesses  the  5  CT  skills 
through  40  multiple-choice  items.  Published  research  relying  on  the  WGCTA  is  abundant, 
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Table  L  Definitions  of  WGCTA  Skills 


Critical  Thinking  Skill 

Definition 

1.  Inference 

Discriminating  among  degrees  of  truth  or  falsity  of 
inferences  drawn  from  given  data 

2.  Recognition  of  Assumptions 

Recognizing  unstated  assumptions  or 
presuppositions  in  given  statements  or  assertions 

3.  Deduction 

Determining  whether  certain  conclusions 
necessarily  follow  from  information  in  given 
statements  or  premises 

4.  Interpretation 

Weighing  evidence  and  deciding  if  generalizations 
or  conclusions  based  on  the  given  data  are 
warranted 

5.  Evaluation  of  Arguments 

Distinguishing  between  arguments  that  are  strong 
and  relevant  and  those  that  are  weak  or  irrelevant  to 
a  particular  question  at  issue 

Source:  Data  adapted  from  Goodwin  Watson  &  Edward  M.  Glaser,  Watson-Glaser  Critical 
Thinking  Appraisal:  Manual  (San  Antonio,  TX:  Psychological  Corporation,  1980),  2. 


addressing  the  importance  of  CT  in  career  fields  to  include  emergency  management,  nursing, 
education,  and  intelligence.16 

While  the  WGCTA  itself  is  broken  into  the  five  CT  skills,  the  individual  test  results  yield 
three  categories:  (1)  recognize  assumptions,  (2)  evaluate  arguments,  and  (3)  draw  conclusions. 
Factor  analysis  revealed  a  more  repeatable  and  reliable  assessment  by  combining  inference, 
deduction,  and  interpretation  into  the  category  of  draw  conclusions.  As  a  new  category  not 
defined  in  Table  1,  drawing  conclusions  is  the  act  of  “arriving  at  conclusions  that  logically 
follow  from  the  available  evidence.”17 

Professional  literature  as  well  as  the  research  reported  in  the  test  manual  established  the 


psychometric  qualities  of  reliability  and  validity  for  the  WGCTA.18  For  internal  consistency 
reliability  coefficients  and  Standard  Errors  of  Measurement,  the  WGCTA  scored  a  0.83  and  2.63, 
respectively.19  The  two  versions  of  the  WGCTA  available  for  pre-  and  post-testing  options  in 
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educational  and  developmental  programs  provided  split-half  reliability  as  well.  Watson  and 
Glaser  examined  the  WGCTA’s  validity  in  several  settings  with  different  populations.  Despite 
the  various  definitions  of  CT,  students  at  various  levels  and  across  several  lines  of  study 
performed  on  the  WGCTA  in  a  manner  to  lend  criterion  validity  to  the  multiple  attempts  to 
develop  CT  skills  in  any  environment.20  Watson  and  Glaser  assessed  the  construct  validity, 
including  content  validity,  internal  factor  structure,  and  convergent  and  discriminate  validity, 
with  supportive  results.21  The  established  psychometric  qualities  of  the  WGCTA  make  it  a 
useful  measuring  instrument  for  research  and  for  programs  exploring  the  development  of  CT. 

Given  the  amount  of  time  and  research  required  to  create  and  validate  a  survey 
instrument,  the  military  should  use  an  existing  tool  in  order  to  measure  CT.22  The  AF  must 
remember  the  WGCTA  is  a  single  assessment,  and  is  not  suitable  as  the  sole  metric  for 
identifying  critical  thinkers.  Some  critical  thinkers  will  possess  different  modalities  of  thinking 
that  does  not  effectively  translate  to  the  WGCTA’s  measurement.  As  with  any  assessment  of 
Airmen,  the  AF  must  consider  the  supervisor’s  assessment  and  the  individual’s  performance. 
Improving  Critical  Thinking 

Upon  measuring  CT  in  a  population,  several  participants  will  likely  want  to  explore 
different  ways  to  improve  those  skills.  This  is  a  legitimate  endeavor  for  all  Airmen  as  CT  skills 
are  not  static.23  One  study  comparing  the  development  of  CT  skills  across  different  age  groups 
found  that,  “adult  students  do  not  appear  to  be  dramatically  different  from  their  younger 
counterparts  in  terms  of  their  reflective  thinking,  including  their  epistemic  assumptions  and  the 
way  they  justify  their  beliefs  in  the  face  of  uncertainty.”24  The  development  of  CT  should  not  be 
limited  to  just  the  brand-new  officers  and  enlisted  on  the  flightline  nor  to  the  strategic-level 
thinkers  in  the  Pentagon. 
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The  importance  of  selecting  the  right  faculty.  When  creating  a  CT  program,  the 
organization  must  know  which  individuals  are  critical  thinkers  before  determining  the  faculty. 
Magnussen’s  research  in  a  nursing  program  suggested  the  CT  skills  of  graduating  students 
correlated  with  the  CT  skills  of  the  instructors,  even  to  the  point  of  fault.25  Students  with  low 
scores  improved  to  approximate  the  instructors’  CT  scores  and  students  with  scores  already 
similar  to  the  faculty  remained  roughly  the  same.  The  worrying  portion  of  the  research  was  the 
fact  that  the  students  initially  scoring  high  in  CT  skills  dropped  and  became  average  through  the 
course  of  the  multi-year  program.  Per  Blondy,  significantly  higher  CT  skills  for  a  nursing  school 
faculty,  when  compared  to  the  students,  was  critical  to  the  success  of  CT  development.26  In  a 
similar  study,  there  were  parallel  themes  in  the  difference  of  CT  skills  between  uniformed  police 
and  police  cadets.27  Finally,  tutors  in  a  successful  WGCTA  test  preparation  program  for  teachers 
scored  significantly  higher  than  the  students  did.28  The  significant  differences  between  the 
instructor  and  student  scores  in  these  studies  suggested  a  successful  program  requires  a  faculty 
with  strong  CT  skills,  and  additional  research  conducted  as  part  of  this  project  suggested  the 
talent  for  a  successful  program  was  already  in  existence  at  AU  (see  Implications). 

The  flexibility  of  the  human  mind.  CT  is  not  a  static  item  such  as  one’s  intelligence 
quotient.  Instead,  people  can  improve  CT  skills  at  any  age.29  Conversely,  CT  skills  are  also 
perishable  and  can  deteriorate  if  the  individual  does  not  dedicate  oneself  to  the  maintenance  and 
improvement  over  time.30  Kegan  and  Lahey  took  a  comprehensive  approach  to  the  development 
of  the  adult  mind,  including  CT,  and  identified  three  general  plateaus  for  adults:  (1)  socialized 
mind,  (2)  self-authoring  mind,  and  (3)  self-transforming  mind.31  The  socialized  mind  is  where 
most  people  spend  their  lives,  capable  of  engaging  in  conversation  and  arguments.  The  self¬ 
authoring  mind  is  able  to  accumulate  new  information  from  a  variety  of  sources  and  build  new 
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approaches  to  various  challenges  by  examining  the  existing  processes.  While  somewhat  easy  to 
attain  for  most  people,  this  still  requires  continued  effort  to  stay  on  this  plateau  and  not  regress  to 
the  socialized  mind  when  external  pressures,  such  as  a  deadline,  approach.  The  self¬ 
transforming  mind  constantly  seeks  new  ways  to  solve  problems  that  others  may  not  even  realize 
exist.  Few  will  ever  reach  this  highest  level  and  those  that  do  must  constantly  strive  to  remain  on 
that  plateau.32  While  Kegan  and  Lahey’s  theory  was  more  expansive  than  CT  alone,  it  indicated 
the  fluid  nature  of  one’s  comfort  with  mental  complexity. 

Focusing  specifically  on  CT,  Reed  explored  the  potential  to  develop  CT  skills, 
concluding,  “students  in  the  experimental  group  performed  at  a  statistically  significantly  higher 
level  than  students  in  the  control  group.”33  Reed  found  both  younger  (less  than  age  22)  and  older 
(over  age  22)  students  benefitted  equally  from  CT  training.  As  a  complement  to  Moore’s 
research34  indicating  life  experience  does  not  necessarily  directly  correlate  with  improved  CT 
skills,  Reed  found  that  “age  and  gender  do  not  appear  to  play  significant  roles  in  developing 
college  students’  critical  thinking  abilities.”35  Peerbolte  studied  the  CT  skills  of  disaster 
management  professionals,  finding  “no  correlation  between  a  participant’s  score  and  the 
dependent  variables  of  age  and  gender. .  .but  positive  correlation  between  a  participant’s  score 
and  the  independent  variables  of  years  of  education  and  years  in  occupation.”36 

The  AF  mission  requires  personnel  capable  of  recognizing  personal  thought  processes 
and  making  structured  and  reasoned  analysis  to  reach  decisions.  Research  supports  that  the  AF 
can  purposely  develop  CT,  meeting  the  AF  FOC’s  requirements.37  Programs  supporting  CT 
development  already  existed  around  the  AF  in  limited  capacity,  but  these  programs  were 
typically  limited  to  a  particular  set  of  career  fields.38  A  structured  holistic  approach  will  be 
critical  to  integrate  CT  improvement  programs  into  several  forms  of  PME,  both  officer  and 
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enlisted.  In  building  CT  into  PME  curriculum,  it  would  be  desirable  to  measure  CT  objectively 
through  a  validated  survey  instrument  and  to  educate  faculty  and  mentors  on  educational 
processes  for  fostering  CT  skills. 

Considerations  when  building  the  critical  thinking  program.  Multiple  programs 
already  existed  across  academia  to  build  CT  skills  in  various  disciplines  such  as  organization 
leadership  and  nursing,  with  several  organizations  publishing  outlines  of  the  training  programs  as 
well  as  results.  Elder  and  Paul  insisted  critical  thinkers  must  “routinely  apply  intellectual 
standards  to  the  elements  of  reasoning  in  order  to  develop  intellectual  traits.”39  Kiltz  further 
elaborated  with  the  assessment  “to  develop  critical  thinking  skills,  students  must  be  active 
learners  in  the  learning  process  and  they  must  be  required  to  identify  and  solve  unstructured 
problems  using  multiple  information  sources.”40  Paul  and  Elder  even  identified  10  intellectual 
standards,  8  elements  of  reasoning,  and  8  intellectual  traits,  ultimately  developing  35  dimensions 
of  critical  thought.41  In  essence,  the  AF  needs  to  apply  structured  problem  solving  at  PME  in 
order  to  develop  CT,  generating  warfighters  able  to  operate  effectively  in  an  ambiguous 
environment. 

The  National  Council  for  Excellence  in  CT  (NCECT)  has  provided  tailored  CT 
development  programs  to  schools  and  businesses  for  over  30  years.  Emphasizing  the  need  for 
long-term  sustained  development  of  CT,  business  programs  tended  to  consist  of  five  two-day 
seminars  covering  the  topics  of  (1)  recognizing  the  importance  of  CT,  (2)  using  the  tools  of  CT 
to  make  better  decisions,  (3)  understanding  the  barriers  to  CT,  (4)  learning  the  art  of  analysis, 
and  (5)  learning  the  art  of  assessing  thought.42  The  program  “clarif[ies]  what  is  meant  by  the 
concept  of  critical  thinking  and  develop  [s]  practical  ways  to  infuse  critical  thought  into  our 
professional  work  both  individually  and  institutionally.”43  NCECT’s  website  offered  additional 
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course  structures  for  consideration  in  either  building  an  organization’s  own  CT  program  or  hiring 
a  team  to  visit  the  site  and  conduct  the  training. 

Based  on  various  searches  through  the  AU  portal  as  well  as  ProQuest  and  EBSCOhost, 
very  limited  publically  available  information  suggested  possible  CT  programs  for 
implementation  across  the  Air  Force.44  One  guide  for  the  importance  of  CT  skills  in  students 
tried  to  provide  a  lesson  plan  for  academic  instructors  at  Maxwell  Air  Force  Base,  Alabama  to 
build  CT  awareness;  however,  this  2008  guide  only  reviewed  a  basic  definition  of  CT  before 
discussing  Bloom’s  taxonomy  and  presenting  a  case  study.45  In  short,  the  guide  was  only  an 
introduction  and  insufficient  for  the  development  of  CT  skills.  The  Army  shared  the  concern  of 
poor  development  in  CT  skills  and  claimed  CT  was  a  vital  component  of  effective  mission 
command.46  Likewise,  the  Army  review  of  PME  did  not  reveal  programs  specifically  designed  to 
develop  CT  skills.  Although  the  AF  repeatedly  recognized  the  need  for  CT  development,  no 
single  program  existed  that  supported  a  sustained  education  as  detailed  by  NCECT. 

Understanding  the  concept  of  CT  and  the  composite  skills  does  not  effectively  transition 
to  a  general  awareness  of  an  individual’s  flawed  decision-making.  Convincing  Airmen  that  they 
need  to  improve  their  methods  and  models  is  a  difficult  task.  People  will  typically  “remain 
convinced  that  what  they  are  doing  is  satisfactory.  Further,  outsiders  who  attempt  to  induce 
change  face  opposition  because  employees  presume  that  external  consultants  are  arrogant  in 
suggesting  that  things  are  not  right,  and  that  change  is  needed.”47  Considering  potential 
application  through  PME,  Paul  discovered  three  disturbing  trends  in  an  assessment  of  CT  across 
multiple  civilian  educational  institutions: 

(1)  Most  college  faculty  at  all  levels  lack  a  substantive  concept  of  critical  thinking. 

(2)  Most  college  faculty  [do  not]  realize  that  they  lack  a  substantive  concept  of  critical 
thinking,  believe  that  they  sufficiently  understand  it,  and  assume  they  are  already 
teaching  students  it. 
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(3)  Lecture,  rote  memorization,  and  (largely  ineffective)  short-term  study  habits  are  still 
the  norm  in  college  instruction  and  learning  today.48 

A  successful  CT  development  program  will  require  senior  leadership’s  understanding  and 

continued  support. 

In  summary,  CT  is  an  obvious  fit  with  PME  and  the  operational  AF  as  it  is  about  problem 
solving  in  ambiguous  situations.  McKown’s  1997  ACSC  study,  the  first  publically  available  Air 
University  product  exploring  CT,  ultimately  concluded  that  “innovative  problem  solving,  critical 
analytical  thought  and  sound  professional  judgment  have  been  and  will  continue  to  be  keys  in 
our  military  leadership  achieving  battlespace  dominance.”49  PME  offers  unique  opportunities  in 
that  Airmen  participate  in  various  forms  of  in-residence  and  distance  learning  programs  at 
multiple  points  across  a  career.50  CT  cannot  just  be  a  matter  of  an  introductory  course  at  the  first 
PME,  but  be  integrated  intentionally  throughout  the  PME  curriculum  in  a  holistic  fashion. 
Finally,  the  AF  should  assess  CT,  both  frequently  and  formally,  throughout  PME  to  determine 
whether  the  programs  are  effective. 

Methodology 

The  purpose  of  the  research  was  to  identify  the  current  state  of  CT  skills  among  ACSC, 
SAASS,  and  AWC  students  in  order  to  create  a  baseline,  and,  using  a  series  of  t-tests,  determine 
any  statistically  significant  differences  between  the  three  samples  of  students.  This  chapter 
covers  the  details  of  the  populations  used  for  the  study,  data  collection,  and  data  analysis. 

Population  and  Sample 

The  intended  population  was  AD  AF  officers,  field  grade  or  above,  with  AD  AF 
residence  students  attending  ACSC,  SAASS,  and  AWC  population  during  AY16  serving  as  a 
convenience  sample.51  The  AF  sends  officers  to  schools  such  as  ACSC  and  AWC  if  the  officers’ 
records  are  in  the  top  20%  of  a  given  year  group.  SAASS  is  a  highly  competitive  advanced 
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studies  program  available  to  officers  as  they  complete  an  IDE  program,  such  as  ACSC.  The  three 
schools  do  not  screen  for  CT  skills  specifically  as  consideration  for  attendance.  The  research 
design  sampled  the  students  when  they  were  between  three  and  four  months  into  the  academic 
programs.  Conducting  data  collection  this  far  into  a  1-year  program  precluded  the  option  of  a 
pre-test  and  post-test  assessment,  exploring  whether  the  schools  developed  CT  skills  within  the 
course  of  the  year.  The  schools’  curriculums  convey  fundamental  concepts  of  CT;  however,  none 
of  the  schools  has  specific  programs  or  courses  designed  specifically  to  build  CT  skills.  While 
the  results  of  this  study  can  only  be  generalized  to  the  top  20%  of  AD  AF  officers,  the  lack  of 
any  CT  screening  as  a  prerequisite  would  suggest  that  the  rest  of  the  AD  AF  officer  population 
would  likely  have  the  same  or  lower  average  scores,  but  not  higher.52 

ACSC  and  SAASS  students  participated  in  the  study  on  a  strictly  voluntary  basis.  While 
highly  encouraged,  AWC  student  participation  was  also  voluntary.  Due  to  the  small  size  and 
heavy  workload  of  SAASS,  the  Dean  solicited  volunteers  who  then  received  the  link  to  take  the 
appraisal  at  their  convenience.  The  researcher  selected  potential  participants  from  ACSC  and 
AWC  through  a  simple  random  sampling  with  replacement53  and  reclaimed  expired  instruments 
as  individuals  chose  not  to  take  the  assessment.  The  researcher  conducted  three  rounds  of  data 
collection  for  ACSC  and  two  rounds  of  data  for  AWC  in  order  to  collect  a  sufficient  sample. 
Results 

A  total  of  133  students  participated  in  the  research  across  the  three  schools.  Table  2 
depicts  the  participation  in  detail.  The  research  compared  the  independent  variable  of  school 
affiliation  (ACSC,  SAASS,  or  AWC)  to  determine  if  there  were  statistically  significant 
differences  in  the  WGCTA  scores  across  the  three  populations  as  well  as  the  overall  score.  The 
research  design  applied  descriptive  statistics  and  t-tests  to  analyze  the  data.  Table  3  identifies  the 
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Table  2.  Participation  by  School 


AD  AF 
Population 

AD  AF 
Participants 

%  of  Participation 

ACSC 

295 

82 

28% 

SAASS 

36 

13 

36% 

AWC 

92 

38 

41% 

Total 

423 

133 

31% 

mean,  standard  deviation,  percentile  ranking,  as  well  as  minimum  and  maximum  scores  for  each 
school.  The  percentile  ranking  was  a  comparison  between  the  scores  of  the  population  compared 
to  the  graduate  degree  normative  group,  consisting  of  “working  adults  from  various  industries, 
occupations,  and  organizational  levels  who  share  a  common  level  of  completed  education...  the 
samples  are  not  limited  to  students  or  recent  graduates.”54  The  graduate  degree  normative  group 
consisted  of  2,321  participants  ranging  across  38  occupations  to  include  entry-level  positions, 
government  service,  and  executive  leadership.55 
Table  3.  WGCTA  Descriptive  Statistics 


Mean 

Standard 

Deviation 

Percentile 

Minimum 

Score 

Maximum 

Score 

ACSC 

27.07 

6.100 

36 

13 

38 

SAASS 

30.92 

4.958 

61 

21 

36 

AWC 

27.42 

6.664 

36 

13 

38 

The  t-test  is  an  inferential  statistical  test  “used  to  determine  whether  two  means  are 
significantly  different  at  a  selected  probability  level.”56  The  t-tests  explored  any  differences 
between  (1)  ACSC  and  AWC,  (2)  ACSC  and  SAASS,  and  (3)  AWC  and  SAASS.  While  there 


were  several  small  differences  between  the  results  of  the  three  schools,  only  the  difference 
between  ACSC  and  SAASS  was  statistically  significant  based  on  a  probability  level  of  0.05.  The 
abbreviated  results  for  all  three  t-tests  are  in  table  4  and  the  full  t-test  results  are  in  appendix  A. 
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Table  4.  Abbreviated  Results  of  T-Tests  for  ACSC,  SAASS,  and  AWC  WGCTA  Scores 


Levene's  Test 
for  Equality  of 
Variances 

t-test  for  Equality  of  Means 

F 

Sig. 

T 

Sig.  (2- 
tailed) 

Mean 

Difference 

Std.  Error 
Difference 

rjl 

c 

ACSC  - 
AWC 

.076 

.783 

-.282 

.778 

-.348 

1.233 

o 

73 

’  S- 

a 

c 

ACSC  - 
SAASS 

1.152 

.286 

-2.162 

.033 

-3.850 

1.781 

r- 

o 

O 

AWC  - 
SAASS 

1.087 

.302 

-1.733 

.089 

-3.502 

2.021 

Note:  difference  considered  significant  if  it  fell  below  the  .05  threshold  in  the  grey  column. 

The  results  as  plotted  on  a  histogram  (see  Figure  1)  suggested  an  even  distribution 
without  significant  kurtosis  but  with  a  slightly  negative  skew.57  As  identified  in  Table  3,  ACSC 
and  AWC  had  very  similar  mean  scores,  min  and  max  scores,  and  standard  deviations.  SAASS 
had  a  higher  mean  score,  less  range  between  the  min  and  max  scores,  and  the  smallest  standard 
deviations  among  the  three  schools. 


Histogram 


Figure  1:  Distribution  of  raw  WGCTA  scores  for  ACSC,  SAASS,  and  AWC  (combined). 
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Discussion 


Applying  t-tests  and  basic  descriptive  statistics,  the  data  supported  the  hypothesis  that 
there  was  no  statistically  significant  difference  in  the  CT  skills  of  AD  AF  students  attending  the 
in-residence  ACSC  and  AWC  programs  in  AY16.  More  specifically,  there  was  no  statistically 
significant  difference  between  the  total  scores  or  across  the  three  individual  skills  of  (1) 
recognizing  assumptions,  (2)  evaluating  arguments,  and  (3)  drawing  conclusions.  However, 
SAASS  scored  significantly  higher  than  ACSC  per  the  t-test  and  reflected  the  smallest  standard 
deviation  across  the  schools.  The  results  plotted  as  a  normal  distribution  without  noteworthy 
kurtosis  and  a  slight  negative  skew.  The  average  score  of  the  ACSC  and  AWC  students  ranked 
at  the  36th  percentile  when  compared  to  the  graduate  degree  normative  group. 

Implications 

In  accordance  with  the  AF  FOC,  CT  is  vitally  important  to  the  success  of  the  AF.  ACSC 
and  AWC  are  a  sample  of  the  top  20%  of  officers  by  their  very  selection  to  attend  IDE  or  SDE 
in-residence.  The  analysis  indicated  the  top  20%  of  AF  officers  at  the  FGO-level  were  below 
average  critical  thinkers  and  were  not  naturally  improving  over  time  because  of  any  educational, 
leadership,  or  operational  experiences.  The  methodology  presented  provides  the  AF  and  DoD 
with  a  way  to  quantitatively  measure  CT,  establish  a  baseline  for  military  personnel,  and 
implement  an  educational  program  where  improvements  in  CT  can  be  clearly  measured  and 
sustained.  This  research  does  not  stop  with  the  small  portion  of  the  AF  surveyed  in  the  research. 
Additional  research  must  begin  building  the  CT  skills  of  the  junior  enlisted  and  officers 
executing  the  tactical  mission.  The  AF  cannot  afford  to  consider  CT  as  an  expectation  or 
privilege  for  the  senior  leadership,  it  is  vital  for  every  Airman  to  begin  or  continue  the  life-long 
pursuit  of  being  a  critical  thinker. 
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Successful  CT  programs  require  strong  critical  thinkers  on  the  faculty.  Although  not  a 
sufficient  sample  size,  six  CT  enthusiasts  from  AU  faculty  and  leadership  volunteered  to  take  the 
WGCTA  as  well.  The  average  raw  score  for  all  participants  was  31.67;  however,  when 
considering  the  possibility  of  building  a  CT  program,  the  lower  2  scores  of  25  and  27  would  be 
excluded,  resulting  in  an  average  raw  score  for  the  remaining  4  of  34.5.  The  new  average  placed 
the  4  participants  in  the  86th  percentile,  higher  than  that  observed  with  the  SAASS  students,  and 
suggested  the  talent  was  already  in  place  to  enhance  CT  integration  for  all  three  schools.  These 
numbers  only  indicated  a  potential,  and  a  more  complete  assessment  will  be  required  before 
identifying  the  right  personnel  to  build  a  CT  development  program. 

Based  on  the  literature  review  and  the  results,  the  AF  needs  to  implement  a  CT 
development  program,  starting  with  faculty  at  ACSC  and  AWC.  This  will  require  first 
identifying  the  strongest  critical  thinkers  as  assessed  by  the  WGCTA,  giving  them  the  time  and 
resources  to  create  a  modified  series  of  seminars  derived  from  the  NCECT’s  recommended 
program,  and  then  begin  sessions  with  all  ACSC  and  AWC  faculty  to  improve  CT  over  a  3- 
month  period  with  quarterly  sessions  thereafter.  The  next  phase  will  entail  applying  those  skills 
to  the  in-class  discussions  through  a  combination  of  integrating  the  faculty  program  materials 
into  the  instruction  and  weaving  measurable  CT  requirements  into  the  syllabi  by  modifying 
existing  case  studies  and  exercises  in-line  with  Kiltz’s  observations.58  CT  should  not  be  a  stand¬ 
alone  block  of  instruction  early  in  the  academic  year,  but  a  periodic  and  recurring  enhancement 
throughout  the  program.  The  faculty  should  make  their  CT  development  program  available  to  the 
rest  of  the  AF  as  a  baseline  and  the  graduates  will  take  their  CT  skills  out  to  the  operational  AF, 
holding  their  personnel  accountable  to  higher  standards  and  further  integrating  CT.  For  AWC 
and  ACSC,  the  recommended  program  is  not  a  matter  of  determining  what  material  to  remove 
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from  the  courses  in  order  to  accommodate  a  CT  program.  Rather,  it  is  how  to  improve  the 
delivery  of  the  existing  materials  in  a  manner  that  fosters  CT  development. 

Areas  for  Further  Research 

To  explore  whether  or  not  PME  improves  CT,  an  additional  study  can  survey  the  AD  AF 
populations  that  graduated  from  AWC  in  AY  15  as  well  as  the  inbound  class  of  AY  17.  With  the 
complete  lack  of  significant  difference  between  ACSC  and  AWC,  coupled  with  previous 
analysis  between  junior  and  mid-level  AF  intelligence  officers,59  additional  research  could 
explore  whether  the  CT  skills  of  Squadron  Officer  School  students  or  even  accessions  and 
technical  school  students  score  any  differently.  Such  a  project  would  be  the  first  study 
expanding  beyond  a  boarded  population  and  would  provide  a  baseline  for  the  general  AF 
population.  A  longitudinal  study  to  track  accessions  throughout  a  career  would  be  a  very 
valuable  and  pure  comparison,  but  would  admittedly  be  very  difficult  to  execute.  The  AU 
Command  Chief  reinforced  the  AF  FOC  and  recommended  the  AF  should  ensure  all  Airmen, 
including  the  enlisted  80%  of  the  force,  have  the  tools  to  refine  their  CT  skills  continuously.60 
For  all  of  the  recommended  studies  above,  future  research  should  collect  additional  demographic 
data  to  look  for  additional  trends  to  include  AFSC,  level  of  education,  schools  attended  (e.g., 
brick  and  mortar,  on-line,  night  school),  and  degrees  held.  Complementing  these  quantitative 
studies,  qualitative  research  should  explore  opportunities  to  integrate  CT  into  PME  more 
effectively,  both  officer  and  enlisted,  and  identify  specific  methods  to  integrate  CT  into  the 
operational  AF. 

Conclusion 

AD  AF  students  attending  ACSC  and  AWC  during  AY2016  collectively  scored  at  the 
36th  percentile  when  compared  to  the  graduate  degree  normative  group.  This  supported  the 
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hypothesis  that  there  was  no  statistically  significant  difference  between  the  CT  skills  of  ACSC 
and  AWC  AD  AF  students.  Through  a  series  of  t-tests,  SAASS  participants  displayed  a 
statistically  significant  improvement  in  scores  than  ACSC,  but  there  was  no  significant 
difference  between  AWC  and  SAASS. 

This  research  was  the  first  of  its  kind,  establishing  a  baseline  against  which  the  AF  could 
assess  the  current  state  of  CT  skills  among  AD  AF  officers.  The  methodology  was  also 
exportable  to  the  rest  of  DoD  for  other  services  determined  to  identify  and  build  critical  thinkers. 
Interested  organizations  in  the  AF  can  also  apply  the  methodology  to  examine  the  development 
of  CT  skills  over  time,  identify  best  practices,  and  continue  to  refine  the  organization’s  approach. 
The  AF  can  measure  and  improve  CT  skills  across  the  force  by  starting  with  a  faculty  program  at 
ACSC  and  AWC,  ultimately  ensuring  a  continuous  emphasis  on  CT  in  both  PME  and  the 
operational  AF. 
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Appendix  A  -  T-Test  Results  for  ACSC,  SAASS,  and  AWC 


T-Test  Results  for  ACSC  and  AWC 


Levene's  Test 
for  Equality  of 
Variances 

t-test  for  Equality  of  Means 

Sig.  12- 
tailed) 

Mean 

Difference 

Std.  Error 
Difference 

95%  Confidence 
Interval  of  the 
Difference 

F 

Sip. 

T 

Df 

Lower 

Upper 

Equal 

variances 

assumed 

.076 

.783 

-.282 

118 

.778 

-.348 

1.233 

-2.789 

2.093 

Equal 

variances 

not 

assumed 

-.273 

66.711 

.786 

-.348 

1.274 

-2.891 

2.195 

T-Test  Results  for  ACSC  and  SAASS 


Levene's  Test 
for  Equality  of 
Variances 

t-test  for  Equality  of  Means 

Sig.  12- 
tailed) 

Mean 

Difference 

Std.  Error 
Difference 

95%  Confidence 
Interval  of  the 
Difference 

F 

Sig. 

T 

Df 

Lower 

Upper 

Equal 

variances 

assumed 

1.152 

.286 

-2.162 

93 

.033 

-3.850 

1.781 

-7.386 

-.314 

Equal 

variances 

not 

assumed 

-2.514 

18.295 

.021 

-3.850 

1.531 

-7.063 

-.637 

T-Test  Results  for  AWC  and  SAASS 


Levene's  Test 
for  Equality  of 
Variances 

t-test  for  Equality  of  Means 

Sig.  12- 
tailed) 

Mean 

Difference 

Std.  Error 
Difference 

95%  Confidence 
Interval  of  the 
Difference 

F 

Sig. 

T 

Df 

Lower 

Upper 

Equal 

variances 

assumed 

1.087 

.302 

-1.733 

49 

.089 

-3.502 

2.021 

-7.563 

.559 

Equal 

variances 

not 

assumed 

-2.002 

27.958 

.055 

-3.502 

1.749 

-7.085 

.081 

Note:  refer  to  “equal  variances  assumed”  row  for  all  three  tests.  All  data  processed  using  SPSS 
vl6. 
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